Computer-aided diagnosis apparatus and computer-aided diagnosis method

ABSTRACT

A Computer-Aided Diagnosis (CAD) apparatus and a CAD method are provided. The CAD apparatus includes an automatic diagnoser configured to perform automatic diagnosis using an image that is received from a probe, and generate diagnosis information including results of the automatic diagnosis. The CAD apparatus further includes an information determiner configured to determine diagnosis information to be displayed among the generated diagnosis information, based on a manual diagnosis of a user, and a display configured to display the received image and the determined diagnosis information.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Patent Application No.10-2014-0178732, filed on Dec. 11, 2014, in the Korean IntellectualProperty Office, the entire disclosure of which is incorporated hereinby reference in its entirety.

BACKGROUND

1. Field

Apparatuses and methods consistent with exemplary embodiments relate toa Computer-Aided Diagnosis (CAD) apparatus and a CAD method.

2. Description of the Related Art

Computer-aided diagnosis (CAD) systems are used in various applicationsto help doctors in diagnosis of diseases. For example, ultrasounddevices are widely used to help doctors diagnose tumors in the breast.Most of the current CAD systems are used by doctors to locate lesions orto determine whether lesions are malignant or benign by reading imagesstored therein.

For more efficient diagnosis, there is a demand for a real-time CADsystem in which locations of lesions and the like may be identified inreal time from videos captured in real time. In the real-time CADsystem, computer diagnosis is performed per frame, and various types ofinformation are displayed on a screen to provide information used bydoctors. However, such a CAD system that helps doctors in real time byproviding all types of information to doctors may hinder diagnosis bydoctors.

SUMMARY

Exemplary embodiments address at least the above problems and/ordisadvantages and other disadvantages not described above. Also, theexemplary embodiments are not required to overcome the disadvantagesdescribed above, and may not overcome any of the problems describedabove.

According to an aspect of an exemplary embodiment, there is provided aComputer-Aided Diagnosis (CAD) apparatus, including an automaticdiagnoser configured to perform automatic diagnosis using an image thatis received from a probe, and generate diagnosis information includingresults of the automatic diagnosis. The CAD apparatus further includesan information determiner configured to determine diagnosis informationto be displayed among the generated diagnosis information, based on amanual diagnosis of a user, and a display configured to display thereceived image and the determined diagnosis information.

The manual diagnosis may include a process for detection of a region ofinterest (ROI) or a process for observation of the detected ROI.

The information determiner may be configured to in response to themanual diagnosis being the process for the detection of the ROI,determine information of the detection of the ROI to be the diagnosisinformation to be displayed, and in response to the manual diagnosisbeing the process for the observation of the detected ROI, determineinformation of the observation of the detected ROI to be the diagnosisinformation to be displayed.

In response to the information determiner determining the information ofthe detection of the ROI to be the diagnosis information to bedisplayed, the display may be configured to display, on the displayedimage, a distinguished marker indicating the detected ROI, based onlocation information of the detected ROI included in the information ofthe detection of the ROI.

In response to the information determiner determining the information ofthe observation of the detected ROI to be the diagnosis information tobe displayed, the display may be configured to display the informationof the observation of the detected ROI based on types and an outputorder of the information of the observation of the detected ROI.

The types and the output order may be determined based on at least oneamong a speed of the probe, an input signal, and a unit of time.

The types may include ROI classification information including a size ofthe detected ROI, characteristics of the detected ROI, and determinationof benignancy or malignancy, Doppler images, ultrasonic elasticityimages, anatomical charts, an examination history of a subject, imagesof body parts of the subject that are acquired by other devices, andinformation of similar cases.

The CAD apparatus may further include a probe speed detector configuredto detect the speed of the probe based on the received image, anddetermine the manual diagnosis based on the detected speed of the probe.

The probe speed detector may be configured to detect the speed of theprobe based on a change in images that are received from the probe, thechange in the images including at least one among a difference in imageintensities of pixels between the images, a difference in histogramsbetween the images, a similarity in the histograms between the images, acorrelation between the images, and a change in information of salientregions of the images.

The automatic diagnoser may be configured to perform the automaticdiagnosis at a same time as the probe speed detector detects the speedof the probe.

The automatic diagnoser may be configured to perform the automaticdiagnosis in response to the probe speed detector determining the manualdiagnosis.

The CAD apparatus may further include an input signal receiverconfigured to receive an input signal from the user, and determine themanual diagnosis based on the received input signal.

According to an aspect of another exemplary embodiment, there isprovided a CAD method, including performing automatic diagnosis using animage that is received from a probe, generating diagnosis informationincluding results of the automatic diagnosis, determining diagnosisinformation to be displayed among the generated diagnosis information,based on a manual diagnosis of a user, and displaying the received imageand the determined diagnosis information.

The determining may include in response to the manual diagnosis being aprocess for detection of an ROI, determining information of thedetection of the ROI to be the diagnosis information to be displayed,and in response to the manual diagnosis being a process for observationof the detected ROI, determining information of the observation of thedetected ROI to be the diagnosis information to be displayed.

In response to the determining the information of the detection of theROI to be the diagnosis information to be displayed, the displaying mayinclude displaying, on the displayed image, a distinguished markerindicating the detected ROI, based on location information of thedetected ROI included in the information of the detection of the ROI.

In response to the determining the information of the observation of thedetected ROI to be the diagnosis information to be displayed, thedisplaying may include displaying the information of the observation ofthe detected ROI based on types and an output order of the informationof the observation of the detected ROI.

The CAD method may further include detecting the speed of the probebased on the received image, and determining the manual diagnosis basedon the detected speed of the probe.

The detecting may include detecting the speed of the probe based on achange in images that are received from the probe, the change in theimages including at least one among a difference in image intensities ofpixels between the images, a difference in histograms between theimages, a similarity in the histograms between the images, a correlationbetween the images, and a change in information of salient regions ofthe images.

According to an aspect of another exemplary embodiment, there isprovided a CAD apparatus, including a diagnoser configured to perform adiagnosis of an image that is received from a probe, a determinerconfigured to determine information to be displayed among results of thediagnosis, based on at least one among a speed of the probe and a numberof input signals that are received from a user, and a display configuredto display the received image and the determined information.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects will be more apparent by describingexemplary embodiments, with reference to the accompanying drawings, inwhich:

FIG. 1 is a block diagram illustrating a Computer-Aided Diagnosis (CAD)apparatus, according to an exemplary embodiment;

FIG. 2 is a block diagram illustrating a CAD apparatus, according toanother exemplary embodiment;

FIG. 3 is a block diagram illustrating a CAD apparatus, according to yetanother exemplary embodiment;

FIGS. 4A and 4B are diagrams illustrating diagnosis information outputon a screen in response to a user's manual diagnosis, according to anexemplary embodiment;

FIGS. 5A and 5B are diagrams illustrating diagnosis information outputon a screen in response to the user's manual diagnosis in FIG. 4B;

FIG. 6 is a flowchart illustrating a CAD method, according to anexemplary embodiment;

FIG. 7 is a flowchart illustrating a CAD method, according to anotherexemplary embodiment; and

FIG. 8 is a flowchart illustrating a CAD method, according to yetanother exemplary embodiment.

DETAILED DESCRIPTION

Exemplary embodiments are described in greater detail below withreference to the accompanying drawings.

In the following description, like drawing reference numerals are usedfor like elements, even in different drawings. The matters defined inthe description, such as detailed construction and elements, areprovided to assist in a comprehensive understanding of the exemplaryembodiments. However, it is apparent that the exemplary embodiments canbe practiced without those specifically defined matters. Also,well-known functions or constructions may not be described in detailbecause they would obscure the description with unnecessary detail.

It will be understood that the terms “comprises” and/or “comprising”used herein specify the presence of stated features or components, butdo not preclude the presence or addition of one or more other featuresor components. In addition, the terms such as “unit”, “-er (-or)”, and“module” described in the specification refer to an element forperforming at least one function or operation, and may be implemented inhardware, software, or the combination of hardware and software.

The exemplary embodiments to be described below may be applied to aComputer-Aided Diagnosis (CAD) apparatus that analyzes ultrasound imagesacquired in real time through a probe to detect regions of interest(ROIs) and to provide diagnosis results on the detected ROIs. However,the exemplary embodiments are not limited thereto, and may also beapplied to CAD apparatuses that receive and diagnose medical imagesacquired by using various techniques, such as Computed Tomography (CT),Magnetic Resonance Imaging (MRI), and the like.

FIG. 1 is a block diagram illustrating a CAD apparatus 100, according toan exemplary embodiment.

Referring to FIG. 1, the CAD apparatus 100 includes an image receiver110, an automatic diagnoser 120, an information determiner 130, and adisplay 140.

The image receiver 110 receives images by using a probe. A user pressesthe probe against an examination region (e.g., abdomen, chest, liver,breast, etc.) of a subject, and moves the probe in various directions toacquire images in real time. The real-time images acquired by the probemay be input to the image receiver 110 in units of frames.

Once the image receiver 110 receives images by using the probe, theautomatic diagnoser 120 uses the received images to perform automaticdiagnosis. The automatic diagnosis includes detection of ROIs fromimages, and determination of the detected ROIs.

The detection of ROIs includes: applying a predetermined automaticdetection algorithm to the received images to detect a location that hasor is suspected to have items of interest such as a lesion; anddetermining an area surrounding the location. Examples of the automaticdetection algorithm may include AdaBoost, Deformable Part models (DPM),Deep Neural Network (DNN), Convolutional Neural Network (CNN), Sparsecoding, and the like.

Further, the determination of an ROI includes: measuring the size of anROI, characteristics of an ROI, such as a shape, an echo pattern, anorientation, a boundary, a texture, an intensity, and the like, orcharacteristics of a lesion according to Breast Imaging Reporting AndData System (BI-RADS) or Liver imaging Reporting And Data System(LI-RADS) lexicon classification; and by using the measured values,determining whether a lesion is malignant or benign.

In an exemplary embodiment, once images are received by using a probe,the automatic diagnoser 120 may generate diagnosis information byperforming automatic diagnosis in parallel with a user's manualdiagnosis. In this case, the diagnosis information may include resultsof each automatic diagnosis, e.g., locations, sizes, characteristics ofitems of interest, BI-RADS characteristics, LI-RADS characteristics, anddeterminations of benignancy or malignancy. Further, the diagnosisinformation may further include Doppler images, ultrasonic elasticityimages, anatomical charts, an examination history of a subject, imagesof a subject's body parts that are acquired by other devices, andinformation on similar cases.

Further, in another exemplary embodiment, once images are received byusing a probe, the automatic diagnoser 120 may perform automaticdiagnosis based on a user's manual diagnosis. In this case, acomputational subject, accuracy and computational complexity of theautomatic diagnosis may vary depending on a user's manual diagnosis.

For example, when a user detects an ROI, the automatic diagnoser 120 mayautomatically detect an ROI in response thereto. Further, when a userobserves the detected ROI, the automatic diagnoser 120 may automaticallydetermine an ROI in response thereto. In this case, the observingprocess of an ROI may be divided into sub-processes, such assegmentation, characteristic extraction, BI-RADS characteristicextraction and classification, and determination of benignancy ormalignancy, and the automatic diagnoser 120 may automatically performsub-processes of a determination process of an ROI that correspond tosub-processes of the observing process of an ROI.

The information determiner 130 may determine diagnosis informationregarding the manual diagnosis that is currently performed by a user andto be output on a screen. In this case, once a user's manual diagnosisis determined, the information determiner 130 may refer to predeterminedinformation as shown in Table 1 below, to determine diagnosisinformation regarding the determined manual diagnosis to be informationto be output on a screen.

The following Table 1 shows an example of predetermined information thatincludes the types of diagnosis information to be output on a screen foreach manual diagnosis of a user; information on whether each type ofdiagnosis information is output, an output time, an output order, andthe like. In this case, the information on whether each type ofdiagnosis information is output, the output time, and the output ordermay be predetermined according to a subject to be diagnosed, a diagnosisobject, a degree of interest of a user, a predetermined unit time, thespeed of a probe, a user's input signal, and the like.

TABLE 1 Information on whether Diagnosis diagnosis Manual information tobe information is Output Output diagnosis output output time orderProcess for Information on Y — — detection of ROI detection of ROIProcess for Information on Y 5 sec. 1 observation of classification ofROI ROI Doppler images N 2 sec. 2 Ultrasonic N 2 sec. 3 elasticityimages Examination Y 3 sec. 4 history Similar cases Y 3 sec. 5

Referring to Table 1 above, when manual diagnosis that is currentlyperformed by a user is a process for detection of an ROI, theinformation determiner 130 may determine information, such as ROIlocation information, which is related to the detection of an ROIdetected by the automatic diagnoser 120, to be diagnosis information tobe output on a screen.

When manual diagnosis that is currently performed by a user is a processfor observation of an ROI, the information determiner 130 may determineinformation on classification of an ROI, an examination history, andsimilar cases, which are indicated as “Y” in terms of whether the typesof information have been output, to be information to be output on ascreen, among information on classification of an ROI, Doppler images,ultrasonic elasticity images, an examination history of a subject, andsimilar cases.

Based on a predetermined output order and output time, the informationdeterminer 130 may determine information on classification of an ROI, anexamination history, and similar cases to be diagnosis information to beoutput in this order for 5 seconds, 3 seconds, and 3 seconds,respectively, during the process for observation of an ROI. In thiscase, even after each type of information is output in sequence, theinformation may be output again when a user performs the process forobservation of the ROI.

As Table 1 above is an example, the present disclosure is not limitedthereto, and a user may sub-divide information on classification of anROI into segmentation information, size information of an ROI,characteristic information, characteristics according to BI-RADS lexiconclassification and results of the classification, characteristicsaccording to LI-RADS lexicon classification and results of theclassification, determination results of benignancy or malignancy, andthe like. Further, a user may determine information on whether each typeof information has been output, an output time, an output order for eachtype of information, and the like. In addition, without setting anoutput time separately for each type of diagnosis information, a usermay determine information to be output for an equal period of time(e.g., one second, two seconds, etc.).

In another exemplary embodiment, the information determiner 130 maydetermine diagnosis information to be output on a screen based on thespeed a probe or a signal input from a user through an interface device.An exemplary embodiment will be described in detail later with referenceto FIGS. 2 and 3.

The display 140 outputs, on a screen, diagnosis information determinedby the information determiner 130 regarding manual diagnosis that iscurrently performed by a user.

For example, once a user detects an ROI, and the information determiner130 determines information on the detection of the ROI to be diagnosisinformation to be output on a screen, the display 140 may display, in animage output on a screen, a distinguished marker that visually indicatesan ROI, by using location information of the ROI included in theinformation on the detection of the ROI. In this case, the distinguishedmarker may be a bounding box, a circle, an oval, a cross, an arrow, andthe like, and may have boundaries with different thicknesses, colors, orthe like, to enable a user to easily identify an ROI.

Further, once a user observes an ROI, the display 140 outputs diagnosisinformation determined by the information determiner 130.

The display 140 may overlay diagnosis information on an image receivedby the image receiver 110, and may sequentially output each type ofdiagnosis information according to an output order of the diagnosisinformation by scrolling the information along with the received image.Alternatively, the display 140 may divide, according to a user' setting,a screen into a first area for outputting a received image and a secondarea for outputting diagnosis information. The display 140 may overlay acurrent image with diagnosis results (e.g., information on detection anddetermination of ROIs) regarding the current image to output thediagnosis results in the first area, and to output diagnosis information(e.g., examination history, similar cases, etc.) to be compared with thediagnosis results regarding the current image in the second area at thesame time as the diagnosis results of the current image or at adifferent time therefrom.

The display 140 may also output, to the second area, diagnosisinformation (e.g., examination history, information on similar cases,etc.) to be compared with the diagnosis information of a current imageat the same time as the diagnosis information of a current image or at adifferent time therefrom.

FIG. 2 is a block diagram illustrating a CAD apparatus 200, according toanother exemplary embodiment.

Referring to FIG. 2, the CAD apparatus 200 includes an image receiver210, an automatic diagnoser 220, an information determiner 230, adisplay 240, and a probe speed detector 250. The image receiver 210, theautomatic diagnoser 220, the information determiner 230, and the display240 are identical to the components 110, 120, 130, and 140 of the CADapparatus 100 illustrated in FIG. 1, such that the descriptions belowwill be focused on the probe speed detector 250.

Once a user moves a probe to acquire images of a subject, the probespeed detector 250 detects the speed of the probe.

In an exemplary embodiment, the probe speed detector 250 may detect thespeed of a probe by using a change in images received by the imagereceiver 210, i.e., by calculating an optical flow from a previous frameto a current frame, or by using a difference image between a previousframe and a current frame.

For example, the probe speed detector 250 may detect the speed of aprobe by using, as a change in images, a difference between the sum ofimage intensities for pixels of a previous image frame and the sum ofimage intensities for pixels of a current image frame acquired through aprobe. That is, once an image frame is acquired by a probe, the probespeed detector 250 may preprocess the image frame to measure an imageintensity for pixels, and may calculate displacement during apredetermined period of time by using the measured image intensity, todetect the speed of a probe based on the calculated displacement.

In another example, the probe speed detector 250 may detect the speed ofa probe based on a difference or similarity between histograms of aprevious image frame and a current image frame. In this case, the probespeed detector 250 may generate histograms of each frame using frequencyof pixel values extracted from the entire area or an area of a frame,and may detect the speed of a probe based on a frequency differencebetween the generated histograms or based on a difference or similarityof histograms when the frequency difference between the generatedhistograms or the similarity of histograms is above a predeterminedvalue.

In yet another example, the probe speed detector 250 may detect thespeed of a probe based on change in information such as information onsalient regions of a previous frame and a current frame.

In another exemplary embodiment, a probe may include a sensor formeasuring a speed, such as a three-axis accelerometer sensor and thelike, and the probe speed detector 250 may use the sensor mounted in theprobe to detect the probe speed.

Once the probe speed is detected, the probe speed detector 250 maydetermine, based on the speed, manual diagnosis which is currentlyperformed by a user. In this case, the probe speed detector 250 maydetermine a user's manual diagnosis by referring to information on theuser's manual diagnosis predetermined for each speed of a probe.

In an exemplary embodiment, Table 2 below shows that the probe speeddetector 250 may compare a detected probe speed to a predeterminedthreshold to determine the speed to be either low or high, and based onthe probe speed, may determine manual diagnosis to be either a processfor observation of an ROI or a process for detection of an ROI.

TABLE 2 Speed Speed threshold of probe (cm/sec.) User's manual diagnosisLow speed Lower than 3 Process for observation of ROI High speed 3 orhigher Process for detection of ROI

In another exemplary embodiment, Table 3 below shows that the probespeed detector 250 may determine the probe speed to be any one of stop,low, or high, and based on the determined probe speed, may determine auser's manual diagnosis to be any one of a process of checkingcomparison information, a process of checking classificationinformation, and a process of detecting an ROI. The process of checkingcomparison information includes checking diagnosis information, such asan examination history of a subject or information on similar cases, incomparison with diagnosis results of a current image. The process ofchecking classification information includes checking, as classificationresults of a current image, segmentation information, the size of anROI, characteristic information, determination of benignancy ormalignancy, and the like.

TABLE 3 Speed Speed threshold of probe (cm/sec.) User's manual diagnosisStop Lower than 0.5 Checking comparison information Low speed 0.5 to 3Checking determination information High speed 3 or higher Detecting ROI

In yet another exemplary embodiment, the probe speed may be subdividedinto a first, a second, . . . , and an n-th step, in which the firststep may be matched to a process for detection of an ROI. Other stepsmay be matched to a process for observation of an ROI, in which theprocess for observation of an ROI is subdivided into a process ofchecking segmentation, a process of checking characteristic information,a process of checking classification results, a process of checkingDoppler images, a process of checking ultrasonic elasticity images, aprocess of checking examination history, a process of checking similarcases, and the like, which are matched to each of the steps.

Once the image receiver 210 receives an image by using a probe, theautomatic diagnoser 220 performs automatic diagnosis using the receivedimage. As described above, the automatic diagnoser 220 may performautomatic diagnosis in parallel with the determination of the probespeed and a user's manual diagnosis, which are determined by the probespeed detector 250. Further, the automatic diagnoser 220 may performautomatic diagnosis in response to a user's manual diagnosis determinedby the probe speed detector 250.

Once manual diagnosis that is currently performed by a user isdetermined according to the speed of a probe, the information determiner230 may determine diagnosis information that corresponds to thedetermined manual diagnosis to be diagnosis information to be output ona screen. In this case, as shown in Table 1, the information determiner230 may determine diagnosis information that corresponds to a user'smanual diagnosis by reference to predetermined information. As describedabove, when a user moves or stops a probe at a low speed for an extendedperiod of time and performs a process for observation of an ROI for anextended duration, the information determiner 230 may determine tooutput each type of the diagnosis information by scrolling theinformation sequentially according to an output order and output time ofthe diagnosis information included in ROI observation information.

The display 240 outputs, on a screen, diagnosis information determinedby the information determiner 230, in which the diagnosis informationcorresponds to manual diagnosis that is currently performed by a user.

Accordingly, a user moves a probe at a high speed so that only theinformation on the detection of an ROI may be displayed on a screen todetermine whether the detected ROI is an ROI to be diagnosed by a user.Further, if the detected ROI is an ROI to be diagnosed, a user may movea probe at a low speed so that information on the observation of an ROImay be output on a screen, in which each type of the diagnosisinformation included in the ROI observation information may be displayedby sequentially scrolling the information, thereby enabling detailedobservation of an ROI.

FIG. 3 is a block diagram illustrating a CAD apparatus 300, according toyet another exemplary embodiment.

Referring to FIG. 3, the CAD apparatus 300 includes an image receiver310, an automatic diagnoser 320, an information determiner 330, adisplay 340, and an input signal receiver 350. In this case, the imagereceiver 310, the automatic diagnoser 320, the information determiner330, and the display 340 are identical to the components 110, 120, 130,and 140 of the CAD apparatus 100 illustrated in FIG. 1, such that thedescriptions below will be focused on the input signal receiver 350.

The input signal receiver 350 may receive a signal input from a user byusing an interface device, and may determine a user's manual diagnosisaccording to the received input signal.

In this case, the interface device may be an external device that ismounted on the CAD apparatus 300, or may be an external device that isconnected through wired or wireless communications, and may include aswitch, a jog shuttle, a joy stick, and the like that may be manipulatedby a body part, e.g., a left hand/a right foot for a right-handed person(and vice versa for a left-handed person), which is not used to operatea probe. However, the interface device is not limited thereto, and mayinclude various devices, such as a hair band type worn on the head, aglasses type worn on the face, a bracelet type worn on a wrist, an anklebracelet type worn on an ankle, a ring type worn on a finger.

For example, Table 4 shows a user's manual diagnosis according to thenumber of signals input from a user within a unit time period. Byreferring to Table 4 below, the input signal receiver 350 may determinemanual diagnosis that is currently performed by a user.

TABLE 4 The number of input signals User's manual diagnosis Once Processfor detection of ROI Twice Process for observation of ROI

Referring to Table 4 above, once a user inputs a signal once within aunit time (e.g., one sec.) through an interface device, the input signalreceiver 350 may determine that a user performs a process for detectionof an ROI. As described above, based on the determination made by theinput signal receiver 350, the information determiner 330 determinesinformation on the detection of an ROI detected by the automaticdiagnoser 320 to be diagnosis information to be output on a screen, andby using ROI location information, the display 340 may overlay adistinguished marker that indicates the location of an ROI on an image.

Then, once a user inputs a signal twice within a unit time through aninterface device, the input signal receiver 350 determines that a userperforms a process for observation of an ROI, and the display 340outputs observation information of an ROI on a screen in the same manneras above.

In this case, as shown in Table 1 above, the information determiner 330may change diagnosis information to be output on a screen during theprocess for observation of an ROI and diagnosis information to be outputon a screen based on an output order and an output time of eachdiagnosis information. Based on the determination made by theinformation determiner 330, the display 340 may sequentially output thedetermined diagnosis information on a screen.

In another example, a user inputs a signal twice to start the processfor observation of an ROI, and then when a signal is input once, eachdiagnosis information included in the ROI observation information may besequentially output. In this case, once the input signal receiver 350receives a signal three times from a user, the display 340 deletesinformation that is currently output on a screen, and may output asubsequent image received by the image receiver 310.

A user's manual diagnosis based on the number of input signals may bechanged in various manners, and a user's manual diagnosis may bepredetermined based on the length or strength of an input signal andaccording to the types of interface devices.

FIGS. 4A and 4B are diagrams illustrating diagnosis information outputon a screen 10 in response to a user's manual diagnosis, according to anexemplary embodiment. FIGS. 5A and 5B are diagrams illustratingdiagnosis information output on the screen 10 in response to the user'smanual diagnosis in FIG. 4B.

As illustrated in FIG. 4A, once manual diagnosis that is currentlyperformed by a user is determined to be a process for detection of anROI, the CAD apparatus 100, 200, or 300 outputs an image 40 receivedfrom a probe on the screen 10, overlays a distinguished marker 41, whichindicates an ROI, on the image 40, and outputs the distinguished marker41. The distinguished marker 41 is not limited to a cross, and may be abounding box, a circle, an oval, an arrow, and the like.

As illustrated in FIG. 4B, once manual diagnosis that is currentlyperformed by a user is determined to be a process for observation of anROI, the CAD apparatus 100, 200, or 300 outputs the image 40 receivedfrom a probe on the screen 10, and overlays classification informationof an ROI on the image 40 that includes a size 42 a of a lesion, aBI-RADS classification result 42 b, segmentation information, lesioncharacteristic information, and the like. In this case, the CADapparatus 100, 200, or 300 may output all the classification informationof an ROI at the same time on a screen, or may sequentially scroll theclassification information based on a predetermined output order and apredetermined output time of the diagnosis information.

FIGS. 5A and 5B are diagrams illustrating an example of ROI observationinformation that may be output on a screen when the CAD apparatus 100,200, or 300 performs a process for observation of an ROI as illustratedin FIG. 4B. Referring to FIGS. 5A and 5B, once it is determined that auser performs a process for observation of an ROI, the CAD apparatus100, 200, or 300 sequentially outputs, on a screen 50, (a) ROIsegmentation information 51; (b) classification information thatincludes a size 52 a of a lesion, a BI-RADS classification results 52 b,and the like; (c) an examination history of a patient, and an image 53of an identical region that is acquired by other devices; and (d)information 54 of similar cases of the patient.

FIG. 6 is a flowchart illustrating a CAD method, according to anexemplary embodiment. FIG. 6 illustrates an example of a CAD methodperformed by the CAD apparatus 100 illustrated in FIG. 1.

In operation 610, the CAD apparatus 100 receives an image from a probe.In this case, the image is acquired through the probe in real time, andmay be input to the CAD apparatus 100 in units of frames.

In operation 620, the CAD apparatus 100 performs automatic diagnosis byusing the received image. The automatic diagnosis may include: a processfor detection of an ROI by applying an automatic detection algorithm tothe image; and a process for classification of the detected ROI.

In operation 630, the CAD apparatus 100 determines diagnosis informationto be output on a screen in response to a user's manual diagnosis thatis currently performed. As shown in Table 1 above, based on a subject tobe diagnosed, a diagnosis purpose, a probe speed, a signal input throughan interface device, and the like, the types of diagnosis information tobe output on a screen, information on whether each type of the diagnosisinformation is output, an output time, an output order, and the like maybe predetermined for each process of the manual diagnosis.

By referring to the predetermined information, the CAD apparatus 100 maydetermine ROI detection information to be diagnosis information to beoutput on a screen when a user's manual diagnosis is a process fordetection of an ROI. Further, when a user's manual diagnosis is aprocess for observation of an ROI, the CAD apparatus 100 may determineROI observation information to be diagnosis information to be output ona screen. In this case, by referring to predetermined information onwhether the ROI observation information is output while the process forobservation of an ROI is performed, the CAD apparatus 100 may determineone or more types of diagnosis information to be output among the ROIobservation information.

In operation 640, the CAD apparatus 100 outputs the received image andthe determined diagnosis information on the screen. For example, once auser performs a process for detection of an ROI, a distinguished markerthat visually indicates an ROI may be displayed in an image output on ascreen by using ROI location information. Further, once a user performsa process for observation of an ROI, the determined diagnosisinformation, e.g., segmentation, the size of an ROI, characteristicinformation, determination of benignancy or malignancy, BI-RADSclassification results, and the like may be overlaid on an image to bescrolled on a screen. In this case, a type of the diagnosis informationmay be displayed in a separate area of a screen without being overlaidon an image, so that a user may easily compare diagnosis results of acurrent image with the diagnosis information.

FIG. 7 is a flowchart illustrating a CAD method, according to anotherexemplary embodiment. FIG. 7 illustrates an example of a CAD methodperformed by the CAD apparatus 200 illustrated in FIG. 2.

In operation 710, the CAD apparatus 200 receives an image from a probe.

In operation 720, the CAD apparatus 200 detects a speed of the probeused by a user to acquire the image. In this case, the probe speed maybe detected by calculating a change in received images, i.e., an opticalflow from a previous frame to a current frame, or by using a differenceimage between a previous frame and a current frame.

For example, the probe speed may be detected by using, as a change inimages, a difference between the sum of image intensities for pixels ofa previous image frame and the sum of image intensities for pixels of acurrent image frame acquired through a probe. Further, the probe speedmay be detected based on a difference or similarity between histogramsof a previous image frame and a current image frame. Alternatively, theprobe speed may be detected based on a change in information of salientregions of a previous frame and a current frame, or by using a speedmeasuring sensor mounted on a probe.

In operation 730, the CAD apparatus 200 determines the user's manualdiagnosis that is currently performed, based on the detected probespeed. In an exemplary embodiment, as shown in Table 2 above, the CADapparatus 200 may determine the detected probe speed to be either highor low, and based on the probe speed, may determine manual diagnosis tobe either a process for observation of an ROI or a process for detectionof an ROI. In another exemplary embodiment, as shown in Table 3 above,the CAD apparatus 200 may determine the speed of a probe to be any oneof stop, low, or high, and based on the determined probe speed, maydetermine a user's manual diagnosis to be any one of a process ofchecking comparison information, a process of checking determinationinformation, and a process of detecting an ROI. In this case, the probespeed and a user's manual diagnosis are not limited to the aboveexemplary embodiments, and may be determined in various manners.

In operation 740, the CAD apparatus 200 performs automatic diagnosisusing the received image. As illustrated in FIG. 7, the automaticdiagnosis may be performed in parallel with the detection of the probespeed in operation 720 and the determination of the user's manualdiagnosis in operation 730. In another exemplary embodiment, automaticdiagnosis may be performed in response to the user's manual diagnosisdetermined in operation 730 based on the detection of the probe speed inoperation 720.

In operation 750, the CAD apparatus 200 generates diagnosis informationthat includes results of the automatic diagnosis. The diagnosisinformation refers to results of diagnosis performed by using a currentimage, includes information on detection of an ROI and classification ofan ROI, and may further include Doppler images, ultrasonic elasticityimages, an examination history of a subject, information on similarcases, and the like.

In operation 760, the CAD apparatus 200 determines diagnosis informationto be output on a screen in response to the user's manual diagnosis,among the generated diagnosis information. For example, when a usermoves a probe at a high speed for the process of detection of an ROI,the information on the detection of an ROI, which is detected byautomatic diagnosis, is determined to be diagnosis information to beoutput on a screen. Further, when a user moves a probe at a low speedfor the process of observation of an ROI, information on classificationof an ROI, Doppler images, ultrasonic elasticity images, an examinationhistory, similar cases, and the like may be determined to be diagnosisinformation to be output on a screen. In this case, if a user performsthe process for observation of an ROI for an extended duration, eachtype of the diagnosis information may be displayed by sequentiallyscrolling the information according to a predetermined output order anda predetermined output time.

In operation 770, the CAD apparatus 200 outputs the received image andthe determined diagnosis information on the screen, in which thedetermined diagnosis information may be overlaid on the received image,or may be output in other areas on the screen where the received imageis not output.

FIG. 8 is a flowchart illustrating a CAD method, according to yetanother exemplary embodiment. FIG. 8 illustrates an example of a CADmethod performed by the CAD apparatus 300 illustrated in FIG. 3.

In operation 810, the CAD apparatus 300 receives an image from a probe.

In operation 820, the CAD apparatus 300 receives an input signal from auser through an interface device. In this case, the interface device isan external device that is mounted in the CAD apparatus 300 or anexternal device that is connected through wired or wirelesscommunications, and may include a switch, a jog shuttle, a joy stick,and the like. However, the interface device is not limited thereto, andmay include a signal generating device that is manufactured in variousmanners, such as a hair band, glasses, a bracelet, an ankle, a ring, andthe like.

In operation 830, the CAD apparatus 300 determines the user's manualdiagnosis that is currently performed, based on the received inputsignal. In this case, a user's manual diagnosis may be predeterminedaccording to the number of input signals as shown in Table 4 above.

Once the received image is output on a screen, a user may input a signalonce to transmit an intention to perform the process for detection of anROI to the CAD apparatus 300. Further, a user may input a signal twicewithin a unit time to transmit an intention to perform the process forobservation of an ROI to the CAD apparatus 300, in which the process forobservation of an ROI includes a process of checking classificationresults of an ROI, and the like. The above exemplary embodiments areillustrative, and one or more exemplary embodiments may also berealized.

In operation 840, the CAD apparatus 300 performs automatic diagnosis byusing the received image. As illustrated in FIG. 8, the automaticdiagnosis may be performed in parallel with the reception of the inputsignal from the user in operation 820 and the determination of theuser's manual diagnosis in operation 830. Alternatively, the automaticdiagnosis may be performed in response to the user's manual diagnosisdetermined in operation 830.

In operation 850, the CAD apparatus 300 generates diagnosis informationthat includes results of the automatic diagnosis. The diagnosisinformation may include information on detection and observation of anROI, in which the ROI observation information may include ROIclassification information, Doppler images, ultrasonic elasticityimages, an examination history of a subject, information on similarcases, and the like.

In operation 860, the CAD apparatus 300 determines diagnosis informationto be output on the screen in response to the user's manual diagnosisamong the generated diagnostic information.

In operation 870, the CAD apparatus 300 outputs the received image andthe determined diagnosis information on the screen.

In addition, the exemplary embodiments may also be implemented throughcomputer-readable code and/or instructions on a medium, e.g., acomputer-readable medium, to control at least one processing element toimplement any above-described exemplary embodiments. The medium maycorrespond to any medium or media which may serve as a storage and/orperform transmission of the computer-readable code.

The computer-readable code may be recorded and/or transferred on amedium in a variety of ways, and examples of the medium includerecording media, such as magnetic storage media (e.g., ROM, floppydisks, hard disks, etc.) and optical recording media (e.g., compact discread only memories (CD-ROMs) or digital versatile discs (DVDs)), andtransmission media such as Internet transmission media. Thus, the mediummay have a structure suitable for storing or carrying a signal orinformation, such as a device carrying a bitstream according to one ormore exemplary embodiments. The medium may also be on a distributednetwork, so that the computer-readable code is stored and/or transferredon the medium and executed in a distributed fashion. Furthermore, theprocessing element may include a processor or a computer processor, andthe processing element may be distributed and/or included in a singledevice.

The foregoing exemplary embodiments are examples and are not to beconstrued as limiting. The present teaching can be readily applied toother types of apparatuses. Also, the description of the exemplaryembodiments is intended to be illustrative, and not to limit the scopeof the claims, and many alternatives, modifications, and variations willbe apparent to those skilled in the art.

What is claimed is:
 1. A Computer-Aided Diagnosis (CAD) apparatus,comprising: a display; and a processor configured to: control todisplay, through the display, an image being received from a probe;perform an automatic diagnosis, using the image being; generatediagnosis information comprising results of the automatic diagnosis;control to detect a user input on the image being displayed; determine amovement speed of the user input that is detected; determine a type of amanual diagnosis, based on the movement speed that is determined;determine a portion of the diagnosis information to be displayed throughthe display, based on the type of the manual diagnosis that isdetermined; and control to display, through the display, the portion ofthe diagnosis information that is determined, the portion of thediagnosis information being superimposed on the image being displayed.2. The CAD apparatus of claim 1, wherein the user input is for a firstprocess for detection of a region of interest (ROI) or a second processfor observation of the ROI that is detected, and the portion of thediagnosis information is further determined based on the user input thatis detected.
 3. The CAD apparatus of claim 2, wherein, based on the userinput that is detected being for the first process for the detection ofthe ROI, the portion of the diagnosis information is determined toinclude first information regarding the detection of the ROI, and basedon the user input that is detected being for the second process for theobservation of the ROI that is detected, the portion of the diagnosisinformation is determined to include second information regarding theobservation of the ROI that is detected.
 4. The CAD apparatus of claim3, wherein the first information regarding the detection of the ROIincluded in the portion of the diagnosis information being displayed ishighlighted relative to the image being displayed.
 5. The CAD apparatusof claim 3, wherein the second information regarding the observation ofthe ROI that is detected included in the portion of the diagnosisinformation being displayed is changed based on a type of theobservation of the ROI that is detected and an output order of theobservation of the ROI that is detected.
 6. The CAD apparatus of claim5, wherein the processor is further configured to determine the type ofthe observation of the ROI that is detected and the output order of theobservation of the ROI that is detected, based on one or more of themovement speed that is determined and a number of the user input that isdetected.
 7. The CAD apparatus of claim 5, wherein the type of theobservation of the ROI that is detected comprises ROI classificationinformation indicating a degree of benignancy or malignancy of the ROIthat is detected.
 8. The CAD apparatus of claim 1, wherein the movementspeed of the user input that is detected is determined based on a changeof the image being displayed, and the change of the image comprises anyone or any combination of a first difference between a first imageintensity of a first pixel in the image, a second difference between afirst histogram of a previous frame in the image and a second histogramof a current frame in the images, a similarity between the firsthistogram and the second histogram and a change of information ofsalient regions of the image.
 9. The CAD apparatus of claim 1, whereinthe processor is further configured to perform the automatic diagnosiswhile detecting the user input.
 10. The CAD apparatus of claim 1,wherein the type of the manual diagnosis comprises a first process fordetection of a region of interest (ROI) in a range and a second processfor observation of the ROI that is detected.
 11. The CAD apparatus ofclaim 1, wherein the movement speed of the user input that is detectedis determined based on data that is received via an accelerometer sensorof the probe.
 12. A Computer-Aided Diagnosis (CAD) method, comprising:displaying an image being received from a probe; performing an automaticdiagnosis, using the image being displayed; generating diagnosisinformation including results of the automatic diagnosis; detecting auser input on the image being displayed; determining a movement speed ofthe user input that is detected; determining a type of a manualdiagnosis, based on the movement speed that is determined; determining aportion of the diagnosis information to be displayed, based on the typeof the manual diagnosis that is determined; and displaying the portionof the diagnosis information that is determined, the portion of thediagnosis information being superimposed on the image being displayed.13. The CAD method of claim 12, wherein the user input is for a firstprocess for detection of a region of interest (ROI) or a second processfor observation of the ROI that is detected, and the portion of thediagnosis information is further determined based on the user input thatis detected.
 14. The CAD method of claim 13, wherein, based on the userinput that is detected being for the first process for the detection ofthe ROI, the portion of the diagnosis information is determined toinclude first information regarding the detection of the ROI, and basedon the user input that is detected being for the second process for theobservation of the ROI that is detected, the portion of the diagnosisinformation is determined to include second information regarding theobservation of the ROI that is detected.
 15. The CAD method of claim 13,wherein the first information regarding the detection of the ROIincluded in the portion of the diagnosis information being displayed ishighlighted relative to the image being displayed.
 16. The CAD method ofclaim 13, wherein the second information regarding the observation ofthe ROI that is detected included in the portion of the diagnosisinformation being displayed is changed based on a type of theobservation of the ROI that is detected and an output order of theobservation of the ROI that is detected.
 17. The CAD method of claim 12,wherein the type of the manual diagnosis comprises a first process fordetection of a region of interest (ROI) in the image and a secondprocess for observation of the ROI that is detected.
 18. Anon-transitory computer-readable storage medium storing instructionsexecutable by a processor of an electronic device to cause the processorto perform a method, the method comprising: displaying an image beingreceived from a probe; performing an automatic diagnosis, using theimage being displayed; generating diagnosis information includingresults of the automatic diagnosis; detecting a user input on the imagebeing displayed; determining a movement speed of the user input that isdetected; determining a type of a manual diagnosis, based on themovement speed that is determined; determining a portion of thediagnosis information to be displayed, based on the type of the manualdiagnosis that is determined; and displaying the portion of thediagnosis information that is determined, the portion of the diagnosisinformation being superimposed on the image being displayed.